1 Introduction

This document reports the code for the discovery of COVID-19 subgroups by symptoms and comorbidities, evaluated on the nCov2019 open dataset. This document, as well as the supporting functions required for its execution, are available in the COVID-19-SDE-Tool GitHub repository.

The COVID-19 infectious disease has led since December 2019 to a worldwide pandemic which is still under control measures. Researchers worldwide are making huge efforts aiming to a comprehensive understanding of the COVID-19 and related healthcare treatments. This work shows the preliminary results of a Machine Learning (ML) approach to identify subgroups of COVID-19 patients based on their symptoms and comorbidities, aiming to a better understanding of variability of severity patterns. In this work, we particularly address the variability (or heterogeneity) between distinct sources populating the research repositories, given the potential impact that this variability may have in data science and the generalization of its results.

We analyzed the raw nCov-2019 dataset release at 2020-05-11. The nCoV2019 dataset comprises a collection of publicly available information on worldwide cases confirmed during the ongoing nCoV2019 outbreak. We included those cases were at least one symptom and an outcome were available. Then, we fixed duplicates and homogenized values in outcomes, comorbidities and symptoms. We mapped the latter to ICD-10 terms. The final sample included 170 cases.

Then, we applied a Multiple Correspondence Analysis 3-dimensional embedding of symptoms and outcomes and a hierarchical clustering. The proper number of clusters for both age-independent and age group analyses were selected by supervised inspection of group consistency.

We found clinically meaningful patient subgroups based on symptoms and comorbidities for specific age groups and age-independent analyses. However, the two most prevalent source countries were divided into separate subgroups with different manifestations of severity.

If you use this code please cite:

Carlos Sáez, Nekane Romero, J Alberto Conejero, Juan M García-Gómez. Potential Biases in COVID-19 Machine Learning due to Data Source Variability. Submitted.

If you are interested in collaborating in this work please contact us.

For further exploration of the results please visit our COVID-19 Subgroup Discovery and Exploration Tool (COVID-19 SDE Tool).

2 Setup

Install and load the required packages.

Source the required functions. These can be found at the COVID-19-SDE-Tool GitHub repository. We recommended to download the whole repository, which includes this document .Rmd file.

3 Data loading

Load the nCov2019 dataset at 2020-05-11. Check the variables set if you plan to use other versions of the dataset.

Convert and derive variables.

Select subdataset with non-missing symptoms and outcomes (assuming missing chronic diseases mean an absence of them).

4 Data preparation

Includes the semantic processing of variables through homogenization of clinical terms, and additional data filtering.

4.1 Semantic preprocessing of symptoms, comorbidities and outcomes

4.1.3 Preprocessing of list of symptoms

# split symptoms texts into vector of (yet unprocessed) symptoms
symptomsLists = sapply(data_symptoms_outcome$symptoms, strsplit, "\\,|;|:|and")
# trim blank spaces
symptomsLists = lapply(symptomsLists, str_trim)
# to lowercase
symptomsLists = lapply(symptomsLists, tolower)
# replace blank spaces with underscore
symptomsLists = lapply(symptomsLists, str_replace_all, " ", "_")

# single replacings of synonyms
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<loss_of_apetite\\>", "anorexia", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<pleuritic_chest_pain\\>", "chest_pain", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<diarrhoea\\>", "diarrhea", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<cardiac_arrythmia\\>", "arrythmia", x))

# multiple replacings of synonyms
# RESPIRATORY 
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<acute_rhinitis\\>|\\<acute_pharyngitis\\>|\\<nasal_congestion\\>|\\<nasal_discharge\\>|\\<rhinhorrea\\>|\\<coryza\\>|\\<coriza\\>|\\<runny_nose\\>|\\<running_nose\\>|\\<pharyngeal_dryness\\>|\\<pharyngeal_discomfort\\>|\\<sore_throat\\>|\\<colds\\>|\\<dysphagia\\>|\\<dry_throat\\>|\\<throat_discomfort\\>|\\<pharyngalgia\\>|\\<flu_like_symptoms\\>|\\<cold\\>|\\<colds\\>", "acute_nasopharyngitis", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<coughing|.+cough", "cough", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<acute_respiratory_distress\\>|\\<acute_respiratory_distress_syndrome\\>|<acute_respiratory_disease_syndrome\\>", "ards", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<difficulty_breathing\\>|\\<shortness_of_breath\\>|\\<chest_distress\\>|\\<chest_myalgia\\>|\\<chest_tightness\\>|\\<chest_discomfort\\>\\<chest_fatigue\\>|\\<respiratory_problems\\>|\\<gasp\\>|\\<grasp\\>|\\<breathing_difficulty\\>|\\<respiratory_complaints\\>", "dyspnea", x)) 
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<sputum\\>|\\<little_sputum\\>|\\<phlegm\\>|\\<little_expectoration\\>", "expectoration", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<severe_acute_respiratory_infection\\>|\\<severepneumonia\\>", "pneumonia severe_pneumonia", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<pneumonitis\\>|\\<mild_pneumonia\\>|\\<lesions_on_chest_radiographs\\>", "pneumonia", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<acute_respiratory_failure\\>|\\<hypoxia\\>|\\<hypercapnia\\>", "respiratory_failure", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<acute_respiratory_disease\\>", "unspecified_respiratory_disease", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<sepsis\\>", "\\<septic_shock\\>", x))
# NON RESPIRATORY
symptomsLists = lapply(symptomsLists, function(x) gsub("fever.+|\\<chills\\>|.+chill|\\<sensation_of_chill\\>", "fever", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<acute_renal_failure\\>|\\<prerenal_failure\\>|\\<kidney_failure\\>", "acute_kidney_injury", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<eye_irritation\\>|\\<red_eye\\>", "conjunctivitis", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<joint_pain\\>|\\<joint_tenderness\\>", "arthralgia", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<dizziness\\>|\\<transientfatigue\\>|\\<transient_fatigue\\>|\\<discomfort\\>|\\<body_malaise\\>|\\<exhaustion\\>|\\<feeling_ill\\>|\\<general_malaise\\>|\\<general_weakness\\>|\\<lack_of_energy\\>|\\<lethargy\\>|\\<malaise\\>|\\<systemic_weakness\\>|\\<iredness\\>|\\<general_weakness\\>|\\<weak\\>|\\<weakness\\>|\\<weaknessness\\>|\\<fatigure\\>", "fatigue", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("+.myalgia|\\<myalgias\\>|\\<aching_muscles\\>|\\<backache\\>|\\<bone_pain\\>|\\<body_ache\\>|\\<milagia\\>|\\<mialgia\\>|\\<muscle_soreness\\>|\\<muscular_soreness\\>|\\<muscle_pain\\>|\\<musculoskeletal_pain\\>|\\<muscular_stiffness\\>|\\<pain\\>", "myalgia", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<nausea\\>|\\<vomiting\\>|\\<vomits\\>|\\<emesis\\>", "nausea_vomiting", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<drowsiness\\>|\\<somnolence\\>|\\<obnubilation\\>", "altered_conciousness_mild", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<congestive_heart_failure\\>|\\<myocardial_dysfunction\\>", "heart_failure", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<infarction\\>|\\<myocardial_infarction\\>|\\<acute_myocardial_infarction\\>", "acute_coronary_syndrome", x))
# symptomsLists = lapply(symptomsLists, function(x) gsub("\\<asymptomatic\\>|\\<none\\>", "NA", x))
symptomsLists = lapply(symptomsLists, function(x) gsub("\\<asymptomatic\\>|\\<none\\>|\\<afebrile\\>", "", x))

# deletion of not informative texts

symptomsLists = lapply(symptomsLists, function(x) gsub("\\<19_related_symptoms\\>|\\<covid\\>|\\<severe\\>|\\<between_others\\>|\\<mild\\>|\\<moderate\\>|\\<hypertension\\>", "", x))
#Note: further use of modulators could be revised
#Note: patients with severe only have that value

symptomsLists = lapply(symptomsLists, function(x) x[sapply(x, str_length)>0])

# join again in a new column for automatic word vector creation
data_symptoms_outcome$symptoms2 = sapply(symptomsLists, paste, collapse  = " ")

4.3 Selection of Age group [!]

To continue with the remaining code please select one of the following chunks of code to execute depending on which Age group you want to analyze. These chunks include the selection of a subset of data for the corresponding age, and setting k as the best number of clusters for each group after a supervised expert review. For the following results we ran the “All ages” group.

Analysis with all ages:

Analysis with ages >65 years:

Analysis with ages between 50-64 years:

Analysis with ages between 18-49 years:

5 Subgroup discovery

Prepare data for analysis, joining symptoms and comorbidities cin a single data.frame and selecting a set of metadata to complement results.

6 Results visualizations

In the following results the information about the subgroups is consistent within scatter plots, histograms and detailed table, and correspond to the subgroups found in the previous application of clustering to the selected Age group with the corresponding number of clusters k.

Perform first some preparation to facilitate visualizations.

6.1 Case embeddings in 2D and 3D scatters

6.1.1 Subgroups

6.1.2 Outcome

6.1.3 Country

6.2 Symptoms, comorbidities and age plots by subgroup

Calculate required statistics.

alphaci = 0.05
uniqueGroups = unique(resultsClustering$groups)
nSubgroups = length(uniqueGroups)
resultsBySubgroup = vector("list",length(uniqueGroups))
# colnames(data_analysis) = substr(colnames(data_analysis),3,nchar(colnames(data_analysis)))

for (i in 1:length(uniqueGroups)){
  
  patientGroupIdx = resultsClustering$groups %in% uniqueGroups[i]
  nPatientsGroup = sum(patientGroupIdx)
  
  data_analysis_subgroup = data_analysis[patientGroupIdx,,drop = FALSE]
  nind = nrow(data_analysis_subgroup)
  resultsSymptoms = sapply(data.frame(data_analysis_subgroup[,1:ncol(symptomsVector), drop = FALSE]), function(x) pCI(x, alphaci)*100)
  colnames(resultsSymptoms) = str_replace_all(colnames(resultsSymptoms),"\\.", " ")
  resultsSymptomsErr = resultsSymptoms[3,] - resultsSymptoms[1,]
  
  resultsComorbidities = sapply(data.frame(data_analysis_subgroup[,-(1:ncol(symptomsVector)), drop = FALSE]), function(x) pCI(x, alphaci)*100)
  colnames(resultsComorbidities) = str_replace_all(colnames(resultsComorbidities),"\\.", " ")
  resultsComorbiditiesErr = resultsComorbidities[3,] - resultsComorbidities[1,]
  
  # sex, age, recovered statistics
  data_analysis_metadata_subgroup = data_analysis_metadata[patientGroupIdx,, drop = FALSE]
  femaleStats = pCI(as.character(data_analysis_metadata_subgroup$sex) %in% 'female',alphaci)*100
  ageMean = mean(data_analysis_metadata_subgroup$ageNum)
  ageErr = qnorm(0.975)*sd(data_analysis_metadata_subgroup$ageNum)/sqrt(nPatientsGroup)
  ageStats = c(ageMean, ageErr)
  recoveredStats = pCI(data_analysis_metadata_subgroup$Outcome == 'Recovered',alphaci)*100
  
  subgroupResults = list(name = paste("Subgroup", i), symptoms = resultsSymptoms, comorbidities = resultsComorbidities, symptomsErr = resultsSymptomsErr, comorbiditiesErr = resultsComorbiditiesErr, nPatientsGroup = nPatientsGroup, femaleStats = femaleStats, recoveredStats = recoveredStats, ageStats = ageStats)
  resultsBySubgroup[[i]] <- subgroupResults
}

6.2.1 Symptoms

6.2.2 Comorbidities

6.3 Detailed results table

alphaci = 0.05
uniqueGroups = unique(resultsClustering$groups)
resultsTable = data.frame(matrix(nrow = ncol(data_analysis)+6, ncol = length(uniqueGroups)))

colnames(data_analysis) = substr(colnames(data_analysis),3,nchar(colnames(data_analysis)))
colnames(data_analysis) = make.names(colnames(data_analysis), unique = TRUE)
colnames(data_analysis) = str_replace_all(colnames(data_analysis),"_", " ")

# Note: this (whihout make.names) returns an error when same text is in symptoms and comorbidities, a solution might be using a column for names, or avoid repated terms
rownames(resultsTable) <- c(sprintf('No. of individuals (n<sub>total</sub> = %d)', nrow(data_analysis)), colnames(data_analysis), 'Females','Age','Recovered','% valid survival (deceased)','Survival days (deceased)')
colnames(resultsTable) <- paste('Subgroup',uniqueGroups)

for (i in 1:length(uniqueGroups)){
  
  patientGroupIdx = resultsClustering$groups %in% uniqueGroups[i]
  nPatientsGroup = sum(patientGroupIdx)
  
  # comorbidity statistics
  data_analysis_subgroup = data_analysis[patientGroupIdx,,drop = FALSE]
  nind = nrow(data_analysis_subgroup)
  data_analysis_subgroupT = t(data_analysis_subgroup)
  resultsS = sapply(data.frame(data_analysis_subgroup), function(x) pCI(x, alphaci)*100)
  subgroupResultColumn = apply(resultsS, 2, function(x) sprintf('%.2f (%.2f-%.2f)',x[1],x[2],x[3]))
  
  # sex, age, recovered statistics
  data_analysis_metadata_subgroup = data_analysis_metadata[patientGroupIdx,]
  #sexProportions = table(corpusSex[patientGroupIdx])/nPatientsGroup
  femaleStats = pCI(as.character(data_analysis_metadata_subgroup$sex) %in% 'female',alphaci)*100
  # ageMean = mean(data_analysis_metadata_subgroup$ageNum)
  ageStats = mCI(data_analysis_metadata_subgroup$ageNum, alphaci)
  recoveredStats = pCI(data_analysis_metadata_subgroup$Outcome == 'Recovered',alphaci)*100
  survivalStats = mCI(data_analysis_metadata_subgroup$date_death_or_discharge_difsym[data_analysis_metadata_subgroup$Outcome == 'Deceased'], alphaci, na.rm = TRUE)
    
  # compile final result column
  subgroupResultColumn = c(as.character(nPatientsGroup), subgroupResultColumn, sprintf('%.2f (%.2f-%.2f)',femaleStats[1],femaleStats[2],femaleStats[3]), sprintf('%.2f (%.2f-%.2f)',ageStats[1],ageStats[2],ageStats[3]), sprintf('%.2f (%.2f-%.2f)',recoveredStats[1],recoveredStats[2],recoveredStats[3]))
  #names(subgroupResultColumn)[c(1,length(subgroupResultColumn)-1,length(subgroupResultColumn))] <- c('n','Females','Median birth date')
  
  # subgroupResultColumn = c(subgroupResultColumn, sprintf('%.2f',sum(!is.na(data_analysis_metadata_subgroup$date_death_or_discharge_difsym))/nind), sprintf('%.2f', mean(data_analysis_metadata_subgroup$date_death_or_discharge_difsym, na.rm = TRUE)))
  subgroupResultColumn = c(subgroupResultColumn,
                           sprintf('%.2f',100*sum(!is.na(data_analysis_metadata_subgroup$date_death_or_discharge_difsym[data_analysis_metadata_subgroup$Outcome == 'Deceased']))/sum(data_analysis_metadata_subgroup$Outcome == 'Deceased')),
                           # sprintf('%.2f', mean(data_analysis_metadata_subgroup$date_death_or_discharge_difsym[data_analysis_metadata_subgroup$Outcome == 'Deceased'], na.rm = TRUE)))
                           sprintf('%.2f (%.2f-%.2f)',survivalStats[1],survivalStats[2],survivalStats[3]))
  
  resultsTable[i] <- subgroupResultColumn
}

# names(resultsTable) <- cell_spec(names(resultsTable), background = "yellow")

tTable = kable(resultsTable, format = "html", escape = F) %>%
  kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive"), full_width = F) %>%
  pack_rows("Symptoms (%, CI 95%)", 2, 1+ncol(symptomsVector)) %>%
  pack_rows("Comorbidities (%, CI 95%)", 2+ncol(symptomsVector), 2+ncol(symptomsVector)+ncol(chronicsVector)-1) %>%
  pack_rows("Demographics (%|x, CI 95%)", 2+ncol(symptomsVector)+ncol(chronicsVector), 2+ncol(symptomsVector)+ncol(chronicsVector)+1) %>%
  pack_rows("Outcomes (%|x, CI 95%)", 2+ncol(symptomsVector)+ncol(chronicsVector)+2, 2+ncol(symptomsVector)+ncol(chronicsVector)+3+1)

groupColors = brewer.pal(n = ncol(resultsTable), name = "Set2")

for (i in 1:ncol(resultsTable)) {
  tTable = tTable %>% column_spec(i+1, background = groupColors[i], include_thead = TRUE) %>%
    column_spec(i+1, background = "inherit")
}

tTable
Subgroup 1 Subgroup 2 Subgroup 3 Subgroup 4 Subgroup 5 Subgroup 6
No. of individuals (ntotal = 159) 54 53 10 23 10 9
Symptoms (%, CI 95%)
headache 7.41 (0.42-14.39) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 4.35 (-3.99-12.68) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
severe pneumonia 0.00 (0.00-0.00) 3.77 (-1.36-8.90) 20.00 (-4.79-44.79) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 11.11 (-9.42-31.64)
acute coronary syndrome 0.00 (0.00-0.00) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 66.67 (35.87-97.46)
myalgia 9.26 (1.53-16.99) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 4.35 (-3.99-12.68) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
acute kidney injury 0.00 (0.00-0.00) 3.77 (-1.36-8.90) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 50.00 (19.01-80.99) 0.00 (0.00-0.00)
heart failure 0.00 (0.00-0.00) 0.00 (0.00-0.00) 20.00 (-4.79-44.79) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 55.56 (23.09-88.02)
dyspnea 5.56 (-0.55-11.67) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 34.78 (15.32-54.25) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
fatigue 16.67 (6.73-26.61) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 26.09 (8.14-44.03) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
acute nasopharyngitis 22.22 (11.13-33.31) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 8.70 (-2.82-20.21) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
septic shock 0.00 (0.00-0.00) 16.98 (6.87-27.09) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 100.00 (100.00-100.00) 0.00 (0.00-0.00)
respiratory failure 0.00 (0.00-0.00) 33.96 (21.21-46.71) 30.00 (1.60-58.40) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 22.22 (-4.94-49.38)
ards 0.00 (0.00-0.00) 35.85 (22.94-48.76) 60.00 (29.64-90.36) 4.35 (-3.99-12.68) 70.00 (41.60-98.40) 11.11 (-9.42-31.64)
cough 59.26 (46.15-72.36) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 56.52 (36.26-76.78) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
pneumonia 1.85 (-1.74-5.45) 83.02 (72.91-93.13) 60.00 (29.64-90.36) 0.00 (0.00-0.00) 60.00 (29.64-90.36) 88.89 (68.36-109.42)
fever 81.48 (71.12-91.84) 0.00 (0.00-0.00) 0.00 (0.00-0.00) 78.26 (61.40-95.12) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
Comorbidities (%, CI 95%)
other 0.00 (0.00-0.00) 3.77 (-1.36-8.90) 0.00 (0.00-0.00) 13.04 (-0.72-26.81) 0.00 (0.00-0.00) 11.11 (-9.42-31.64)
respiratory 0.00 (0.00-0.00) 5.66 (-0.56-11.88) 0.00 (0.00-0.00) 26.09 (8.14-44.03) 40.00 (9.64-70.36) 0.00 (0.00-0.00)
renal 0.00 (0.00-0.00) 22.64 (11.37-33.91) 0.00 (0.00-0.00) 4.35 (-3.99-12.68) 10.00 (-8.59-28.59) 33.33 (2.54-64.13)
cardiovascular 0.00 (0.00-0.00) 16.98 (6.87-27.09) 0.00 (0.00-0.00) 26.09 (8.14-44.03) 10.00 (-8.59-28.59) 0.00 (0.00-0.00)
metabolic 0.00 (0.00-0.00) 58.49 (45.22-71.76) 10.00 (-8.59-28.59) 47.83 (27.41-68.24) 40.00 (9.64-70.36) 44.44 (11.98-76.91)
hypertension 0.00 (0.00-0.00) 90.57 (82.70-98.44) 0.00 (0.00-0.00) 65.22 (45.75-84.68) 20.00 (-4.79-44.79) 55.56 (23.09-88.02)
na 100.00 (100.00-100.00) 5.66 (-0.56-11.88) 90.00 (71.41-108.59) 4.35 (-3.99-12.68) 60.00 (29.64-90.36) 11.11 (-9.42-31.64)
Demographics (%|x, CI 95%)
Females 33.33 (20.76-45.91) 26.42 (14.55-38.28) 30.00 (1.60-58.40) 30.43 (11.63-49.24) 30.00 (1.60-58.40) 22.22 (-4.94-49.38)
Age 48.19 (43.29-53.09) 68.66 (65.64-71.68) 56.50 (47.83-65.17) 71.65 (66.96-76.34) 69.70 (62.08-77.32) 71.56 (62.38-80.73)
Outcomes (%|x, CI 95%)
Recovered 68.52 (56.13-80.91) 0.00 (0.00-0.00) 10.00 (-8.59-28.59) 8.70 (-2.82-20.21) 0.00 (0.00-0.00) 0.00 (0.00-0.00)
% valid survival (deceased) 76.47 9.43 22.22 100.00 10.00 0.00
Survival days (deceased) 16.00 (11.40-20.60) 14.40 (11.66-17.14) 16.00 (12.08-19.92) 14.33 (11.27-17.39) 14.00 (NA-NA) NaN (NaN-NaN)